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ABSTRACT 

Faculty and student feedback on blended courses is instrumental to improving blended courses and 
programs. The purpose of this article is to describe the process and results of blended learning outcome 
assessment at a large, multi-campus, private university. The outcome measures used in this assessment 
were developed in the context of current literature about best practice in assessment of blended learning 
and were designed to gather feedback from faculty and students on multiple aspects of the blended 
courses. The results suggest that, overall, students and faculty were satisfied with the blended courses. 
The sufficiency of the university’s resources for blended learning emerged as the strongest predictor of 
student satisfaction. Faculty rated their blended courses high in terms of course organization, but low in 
terms of the availability of resources and information about online learning and the quality of pedagogy in 
blended instruction. This university has found great value in the use of student and faculty feedback 
toward the improvement of blended courses and programs. An emphasis has been placed on providing 
additional resources, making the available resources more accessible and offering faculty development 
and training in online and blended pedagogy. 
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I. INTRODUCTION 

The success of integrating blended learning into a university culture depends on the institution’s ability to 
adapt its teaching and administrative strategies and promote collaborations among all stakeholders [1]. 
Assessment becomes a key process in identifying systemic needs for development and implementation of 
such strategies. Garrison and Vaughan [2] write: “Assessment .. .may well be the most pervasive issue in 
designing a deep and meaningful learning experience” (p. 100). As such, meaningful assessment of 
blended learning must be multidimensional and encompass data from direct and indirect measures of 
learning, with information gathered from faculty, students, and other stakeholders. This paper focuses on 
indirect measures of blended learning based on quantitative and qualitative feedback from students and 
faculty on two campuses of the same university. 
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A. The Value of Student Feedback Toward Improving Learning in Blended 
Courses 

The use of student perception-based data to improve higher education has been well documented in the 
literature on blended and other forms of e-leaming. Quinn, Fitch and Youn [3], who have developed a 
construct validity model for assessing e-leaming, stress the importance of studying “ the construct of 
student” (p. 324), which they operationally define as students’ demographic data, learning style and 
satisfaction with the course. Several researchers have developed instruments to assess students’ feelings 
and attitudes about e-leaming. Garrison, Cleveland-lnnes, and Fung [4], investigated role adjustments of 
students new to the online community of inquiry. The study focused the impact of teaching, social and 
cognitive presence on students’ adjustment to the online environment. Richardson and Swan [5] studied 
the relationship between students’ social presence, and their perceived learning and satisfaction with the 
instructor. They found that the more comfortable students were interacting with each other and 
participating in course activities, the greater their satisfaction with the instructor and the higher their self¬ 
rating of learning. Similarly, Shea [6] found that where students report higher levels of teaching presence 
in the online environment (specifically, active directed facilitation and effective instructional design), they 
also report higher levels of online learning community. 

Ozkan and Koseler [7] developed a self-report instrument to measure student satisfaction with a particular 
learning management system (FMS). They were able to identify and validate six dimensions that impact 
“perceived learners’ satisfaction from FMS” (p. 1291) that can be effectively used in the pedagogical 
design and use of learning management systems. These are: students’ attitudes about e-leaming overall; 
students’ perceptions of the respective quality of the instructor; interface system used in e-leaming; 
course content or information; services provided by administrators and staff (e.g., technical support); and, 
finally, supportive issues, which include ethical and legal issues, privacy, plagiarism, intellectual 
property, etc. The authors suggest that for e-leaming to be effective, course design and delivery must take 
these dimensions into account. 

Similarly, Pombo, Foureiro and Moreira [8] evaluated the effectiveness of a collaborative learning 
module in a blended course by assessing the students’ views and opinions about the module. They used 
student ratings of specific elements of the module to improve its pedagogy by addressing both the content 
of the module and the tools used to deliver it. Students’ feedback highlighted the importance of 
incorporating assignments that encouraged learning and critical reflection via peer assessment and 
interaction, using tools that were user-friendly and easy to navigate for both synchronous and 
asynchronous communication. 

Student feedback can also provide important clues regarding higher order learning skills (vs. content 
mastery) developed via e-leaming with implications for pedagogical design of e-leaming courses. For 
example, Fopez-Femandez and Rodriguez-lllera [9] used a mixed quantitative and qualitative approach to 
investigate students’ perceptions and attitudes towards the use of e-portfolios in a blended learning 
environment for the purpose of creating a better fit between the students and this particular strategy of 
teaching. Although the results suggested that use of the e-portfolio did not have a significant impact on 
students’ actual learning, the results did indicate that students expressed positive attitudes towards the use 
of the e-portfolio. The authors note that students’ favorable response to the e-portfolio has important 
implications for students’ ability to develop meta-leaming skills such as autonomous management of their 
learning. 

Griesbaum and Gortz [10] have taken a different approach to gathering student feedback. Rather than 
measure students’ reactions to a particular e-learning module or an e-leaming tool, they assessed students’ 
rating of their own and their peers’ performance on a collaborative learning module. The authors found 
that self- and peer- feedback helped to identify group processes and learning behaviors that students 
display during a collaborative learning task (e.g., preparedness, active contribution to the assignment). 
This information can then be used to make informed instructional decisions in designing group 
assignments and in enhancing productive learning behaviors. 
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In sum, student feedback in an e-learning environment is effective as an indirect measure to assess 
students’ satisfaction with their learning environment. As reflected in the literature cited above, there has 
been extensive exploration and analysis of student feedback on individual elements of the e-leaming 
environment. The current study takes a more holistic approach in assessing students’ experiences in 
blended courses along multiple dimensions. 

B. The Value of Faculty Feedback Toward Improving Learning in Blended 
Courses 

Faculty input into and reflection upon the experience of teaching blended courses is considerable for the 
development of blended courses and programs. Faculty buy-in is essential to the success of blended 
learning, and research can shed light on factors that are associated with faculty’s openness to this format 
of teaching. Through the use of focus groups, Hiltz, Shea, and Kim [11] found that faculty’s motivation 
to engage in online teaching is associated with a variety of factors including the flexibility of time and 
location of instruction, and opportunities for increased personal interaction with students. Young [12] 
asserted that the success of the blended learning format requires that faculty feel in control of their 
blended courses. How faculty members define control seems to be a particularly important question in 
understanding their experience of teaching blended courses. Faculty control of content and format may 
need to be analyzed separately. For example, a faculty member who defines course objectives (content) 
may eagerly accept input from an instructional designer on assessment activities in a blended format that 
may help her know how well students are learning what she intends for them to learn. Indeed, at one 
university that has been highly successful with online education, the philosophy is that “faculty are 
discipline experts and should not have to become web designers or technical gurus. They should be able 
to concentrate on teaching and engaging their students” [13, p.63]. 

Niemiec and Otte [14] used the term ownership rather than control. They stated that “faculty must have 
ownership, must feel that blended learning is pursued through (and not against) their prerogatives, their 
responsibility for the design of instruction” (p.93). The assessment of how faculty members view the 
instruction of blended courses and what faculty members expect and want from their university with 
regard to training, resources and support for teaching in a blended format occurs in a variety of ways. 

In their seminal text, Blended Learning in Higher Education, Garrison and Vaughn [2] presented a series 
of ten open-ended faculty interview questions along with a sampling of comments from interviews of 
faculty teaching blended courses. One semester’s evaluation at the University of Calgary revealed 
planned changes including a more detailed student orientation to blended learning, a clearer alignment 
between assessment activities and learning objectives, and an earlier development of resources and 
activities to provide students with a clearer plan at the start of the course. 

In a similar vein, Patsy Moskal [15] of the University of Central Florida’s Research Initiative for 
Teaching Effectiveness (RITE) described the evaluation of blended learning as dancing with a bear, 
referring to the authors’ favorite Russian parable suggesting that evaluating blended learning requires 
tireless persistence. She stated that evaluation must be both reflective and iterative. Faculty, who are 
surveyed repeatedly, are eager to share their successes and concerns, likes and dislikes in teaching 
blended courses. The information has been used to plan training sessions for faculty on specific 
challenges as well as new faculty orientations and faculty development workshops. 

C. The Current Study: WLP Assessment Model Within Institutional Context 

In spring 2009, the university initiated a cross-campus Web Feaming Project (WFP), whose major focus 
was the development of a blended format for five pilot graduate programs to be implemented in fall 2010. 
In academic year 2009-2010, the WFP was supervised by a university-wide Steering Committee co¬ 
chaired by the Deputy Vice President for Academic Affairs and the University Chief Information Officer. 
The Associate VP for Instructional Technology & Faculty Development became “Team Fead” and was 
assigned responsibility for supervision of the various functional teams involved in the project. 
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One of the most important functional teams within the WFP is the Assessment Team, whose original 
charge was to develop a framework for the assessment of the five graduate programs selected for 
inclusion in the project. The assessment team is characterized by an interdisciplinary collaboration of 
administration and faculty from the university’s two main campuses. In addition to ensuring consistency 
of the blended learning assessment effort with existing university assessment processes, the Assessment 
Team also provides direct support to faculty, analysis of the assessment data, and assessment reports for 
the university community. 

The assessment framework for the pilot blended programs includes: 1) assessing student’s learning 
through direct measures of student learning via embedded course assignments; 2) assessing faculty 
teaching effectiveness using the University of Washington IAS Student Course Evaluation System, Form 
X [16] (administered for all courses in all programs, university-wide, each semester); 3) assessing 
students satisfaction with blended courses utilizing WFP Student Survey; 4) assessing faculty rating of 
course quality via the end-of-semester Faculty WFP Survey, and, 5) discussing overall impressions of the 
blended learning experience and the implications of assessment findings via end-of-semester focus groups 
with blended program faculty and administrators. The charge of the WFP Assessment team pertains to 
items 3, 4 and 5, which are the focus of this paper. 

Thus, this assessment study addresses the following questions: (1) Are students satisfied with blended 
learning at the university? (2) How do faculty rate the quality of blended courses in the participating five 
pilot programs? and (3) What are the implications of the assessment findings? 

II. METHOD 


A. Participants 

The WFP included five graduate programs across two campuses of the university: Nursing Education, 
Accounting and Taxation, Fibrary and Information Sciences, Nurse Educator, and Computer Science. 
This pilot assessment project utilized an availability sampling method, which was drawn from the WFP 
faculty and students, and administrators who were involved in the project during the AY 2010-2011. 

1. Faculty 

Overall, 23 faculty members, most of them full-time faculty, taught 37 blended courses (16 in the fall 
semester and 21 in the spring semester) within the above five programs. Ten of these faculty members 
taught multiple blended courses during the academic year. Table 1 summarizes the number of faculty 
members per program per semester. 


Program 

# of faculty members in the 
fall semester 

# of faculty members in the 
spring semester 

Nursing Education 

2 

2 

Accounting and Taxation 

3 

3 

Library and Information Sciences/ 
School Library Media 

3 

4 

Nurse Educator 

5 

4 

Computer Science 

3 

4 


Table 1. Number of Faculty Members per Program per Semester 


The sampling method yielded thirteen completed faculty surveys in the fall and eleven in the spring, 
representing a response rate of 81% and 65%, respectively. In order to preserve confidentiality, data were 
available in the aggregate only, making it impossible to ascertain which course (or faculty member) was 
associated with a particular response set. 
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2. Students 

Overall, 197 students participated in 16 blended courses in the fall and 218 students participated in 21 
blended courses in the spring. Table 2 summarizes the numbers of students per program per semester. 


Program 

# of students in the fall 

semester 

# of students in the spring 
semester 

Nursing Education 

25 

30 

Accounting and Taxation 

43 

44 

Library and Information Sciences/ School 
Library Media 

51 

64 

Nurse Educator 

62 

66 

Computer Science 

16 

14 


Table 2. Number of Students per Program per Semester 


The sampling method yielded 55 student responses to the survey in the fall and 52 student responses to 
the survey in the spring. Because each student had multiple opportunities to respond to the survey (equal 
to the number of blended courses that the student was enrolled in during each semester) the actual number 
of completed surveys (68 in the fall and 60 in the spring) represents 24.46% and 18.98% response rate, 
respectively. Table 3 summarizes student characteristics along the dimensions of gender, age, 
employment and experience with blended courses, per semester. 


% 

male 

% 

female 

Mean age 

Mean # hours work 
per week 

Mean # of blended 

courses 

Fall semester 18 

(n=68) 

82 

36.04 
(sd= 10.99) 

29.4 (sd=15.37) 

2.53 (sd=. 95) 

Spring semester 15 

(n=60) 

85 

33.87 
(sd= 10.25) 

27.67 (k*=15.04) 

1.83 (sd=. 76) 


Table 3. Students’ Characteristics 


3. Administrators 

Administrators who were involved in the WFP were invited to participate in the focus groups after each 
semester and were recruited using an availability sampling method. The following administrators were 
represented in each focus group: the program’s respective chairs and/or deans, the University Director of 
Assessment, the Associate VP for Instructional Technology & Faculty Development, the university’s 
instructional designers, and members of the WFP assessment team. Two focus groups were held each 
semester, one on each campus. Fall 2010 focus groups included 16 and 8 participants on each campus, 
respectively, and the spring 2011 focus groups included 8 participants on each campus. 

B. Instruments 

1. The WLP Student Survey 

The WFP Student Survey (Appendix A) is an online self-report, mixed quantitative and qualitative 
questionnaire. It consists of 3 parts: Part A gathers demographic data about students. Part B consists of 
26 Fikert-scale items that ask students to rate their experiences in their respective blended courses. These 
items are further divided into factors, respectively assessing: 1) Students’ experiences in blended vs. f2f 
courses; 2) Students’ rating of the overall quality of blended courses at [name of university blinded for 
review], and 3) Students’ rating of the learning helpfulness of specific course components. Each of the 
close-ended ordinal-scale items in this section is followed by an invitation to the students to offer 
qualitative comments in relationship to that question. Part C consists of 3 open-ended questions inviting 
students to reflect on their overall experiences in the blended course. Table 4 presents the range of scores 
for each factor of the WFP Student Survey. 
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Factor 1: Important considerations for students when they choose to participate in a blended 

program (items 6-11) 


Factor Score 

Descriptive Category 

5-8 

Not at all important 

9-13 

Slightly important 

14-16 

Moderately important 

17-20 

Very important 

21-25 

Extremely important 

Factor 2*: The experience of blended courses compared to f2f courses (items 13-17, 29, 30). 

*Higher scores indicate blended courses to be a better experience compared to F2F. 

Factor Score 

Descriptive Category 

6-10 

Much worse 

11-15 

Somewhat worse 

16-20 

Neither worse nor better 

21-25 

Somewhat better 

26-30 

Much better 

Factor 3*: The overall quality of blended courses at the university (items 19-22, 24). 

^Higher scores indicate a higher rating of overall quality of blended courses ’ overall quality. 

Factor Score 

Descriptive Category 

5-8 

Strongly disagree 

9-13 

Moderately disagree 

14-16 

Neither disagree nor agree 

17-20 

Moderately agree 

21-25 

Strongly agree 

Factor 4*: Helpfulness of specific course components (items 25-27; 31-39) 

* Higher scores indicate a higher rating of course components ’ helpfulness. 

Factor Score 

Descriptive Category 

12-22 

Not at all helpful 

23-33 

Slightly helpful 

34-44 

Moderately helpful 

45-54 

Very helpful 

55-60 

Extremely helpful 


Table 4. The WLP Student Survey Range of Scores for Part B Factors 


The WFP Assessment Team developed the WFP Student Survey specifically for assessing students’ 
experiences in blended courses. The instrument is adapted from a student survey questionnaire [2] from 
which it derives its initial content validity (e.g., rating the interaction between students and instructor in 
the course, rating the quality of various course components and rating the goodness of fit between the 
online and in-class learning aspects of the course). Additional validation for content of specific items on 
the survey emanates from the Sloan-C goal for student satisfaction as associated with discussion and 
interaction with instructors and peers, satisfaction with services (e.g., advising, registration, access to 
materials), and orientation to online learning [17]. Finally, assessing students’ various considerations 
when choosing a blended program derives its content validity from Allen, Seaman, & Garrett [18], who 
have documented delivery mode (i.e., campus-dominated vs. blended or online delivery), work, and 
family responsibility to be likely factors in program selection. The WFP Assessment Team modified the 
instrument to include additional items that were relevant to the university. Moreover, items were added to 
rate students’ experiences in the blended course as compared to their experiences in face-to-face courses 
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in order to help the university assess student perceptions of the relative efficacy of these learning 
modalities. 

In an effort to increase the instrument’s content and face validity, the survey was pre-tested twice. A first 
draft of the instrument was shared with faculty members who were scheduled to teach WFP blended 
courses the following academic year and who had developing their respective blended courses. Based on 
their feedback, the instrument’s items and the respective response options were modified to enhance 
content and clarity. For example, items that detail specific components that faculty incorporated into their 
blended courses were added, e.g., media content, textbooks, online discussions, and exams. Furthermore, 
a response option of Not Applicable was added to all items that assess course components, allowing for 
the fact that different courses feature different components. Second, upon uploading the instrument to the 
CampusFabs (formerly StudentVoice) platform, members of the WFP Advisory Committee reviewed the 
online version of the instrument. Adjustments were then made to ensure that the platform was user- 
friendly. Finally, feedback from faculty and WFP Advisory Committee members also helped to reduce 
ambiguity in the language of several items. 

To increase the reliability of the instrument, an effort was made to avoid compounded items without 
unduly lengthening the instrument. All ordinal items on Part B feature a 5-point response option (with the 
exception of items that assess course components and that offer a 6 th response option: NA) for unified 
scoring across all items. Because the instrument assesses several constructs, its reliability was tested 
separately for each factor. Cronbach Alpha coefficients range from .8-.9 for the fall semester and 
from .7-.8 for the spring semester. 

2. The WLP Faculty Survey 

The WFP Faculty Survey (Appendix B) is an online, self-report questionnaire that is based upon the 
California State University: Chico Rubric for Online Instruction (ROI) [19]. The ROl offers a framework 
for addressing the question, “What should a quality online course look like?” This rubric has been used 
and modified at the university with the understanding that blended course design and delivery is a 
developmental process that benefits from continual faculty self-assessment. 

Six categories from the rubric are represented in thirty items: (1) Fearner Support and Resources, (2) 
Online Organization and Design, (3) Instructional Design and Delivery, (4) Assessment and Evaluation of 
Student Fearning, (5) Innovative Teaching with Technology, and (6) Faculty Use of Student Feedback. 
Three response options are presented: baseline, effective, or exemplary. Aside each of these response 
options is a statement defining that particular classification. Table 5 presents the range of scores for each 
category of the WFP Faculty Survey. 


Category 1 - Learner Support and Resources (items 1-5) 

Category Score 

Descriptive Category 

5-8 

Limited 

9-12 

Adequate 

13-15 

Exemplary 

Category 2 - Online Organization and Design (items 6-10) 

Category Score 

Descriptive Category 

5-8 

Fimited 

9-12 

Adequate 

13-15 

Exemplary 

Category 3 - Instructional Design and Delivery (items 11-17) 

Category Score 

Descriptive Category 

7-12 

Fimited 

13-18 

Adequate 

19-21 

Exemplary 
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Category 4 - Assessment and Evaluation of Student Learning (items 18-22) 

Category Score 

Descriptive Category 

5-8 

Limited 

9-12 

Adequate 

13-15 

Exemplary 

Category 5 - Innovative Teaching Technology (items 23-26) 

Category Score 

Descriptive Category 

4-6 

Limited 

7-9 

Adequate 

10-12 

Exemplary 

Category 6 - Faculty Use of Stuc 

lent Feedback (items 27-30) 

Category Score 

Descriptive Category 

4-6 

Limited 

7-9 

Adequate 

10-12 

Exemplary 

Total scoring 

30-50 

Limited 

51-71 

Adequate 

72-90 

Exemplary 


Table 5. The WLP Faculty Survey Scoring Categories 


In spring 2011, the survey was revised to include examples for each item. These examples were modified 
from the Instructional Design Tips for Online Feaming, developed by Joan Van Duzer at Humboldt State 
University, to be used in conjunction with the Rubric for Online Instruction [20]. Specific examples 
relevant to the university were included. Reliability coefficients for the faculty survey were calculated 
separately for each semester, yielding an excellent reliability for this instrument (Cronbach Alpha=.94 
and .97, in the fall and spring semesters, respectively). 

3. Focus Groups 

The purpose of the focus groups was to convey the assessment results to the faculty and to provide an 
opportunity to discuss implications of the findings. Accordingly, three central questions were posed 
during each focus group: (a) What is the faculty’s reaction to the findings? (b) How do these results 
influence faculty’s thinking about their blended courses in subsequent semesters, and (c) What do faculty 
need to implement the improvements they seek in their blended courses? 

C. Procedure 

The university’s Institutional Review Board (IRB) permitted data collection and dissemination, utilizing 
the procedures detailed below. 

1. The WLP Student Survey 

Utilizing the CampusFabs interface, the WFP Student Survey was made available to each student in each 
of the WFP blended courses towards the end of the fall 2010 and spring 2011 semesters. Each instructor 
provided the students with a link to the survey, and students could log onto CampusFabs and complete 
the survey using their individually created verification screens. Each student had an opportunity to 
complete a separate survey for each blended course in which he or she was enrolled in during that 
semester. To preserve anonymity and confidentiality, no faculty, administrator or staff member at the 
university was granted access to data regarding the students’ verification screens. Each faculty member 
had access to aggregate data from his or her course, provided that there were three or more surveys 
completed for that course. Members of the WFP Assessment Team and of the WFP Advisory Committee 
had access to aggregate data across all the courses, which included all completed surveys. 
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2. The WLP Faculty Survey 

The WFP faculty survey was administered via CampusFabs to all faculty members who taught a blended 
course in one of the five WFP programs during the fall or spring semesters of AY 2010-2011. An email 
from the University Director of Assessment on behalf of the WFP Assessment Team requested faculty’s 
participation. To preserve participants’ anonymity and confidentiality, the assessment team had access to 
the results only in the aggregate, not by individual faculty member. The assessment team was not 
provided with the names of the faculty members who completed the survey. 

3. The Focus Groups 

The office of the Associate Vice President for Instructional Technology and Faculty Development 
emailed the WFP faculty to request their participation in focus groups at the end of each semester. The 
AVP and members of the WFP Assessment Team facilitated the 90-minute group discussions. 

D. Design 

The WLP assessment of blended learning utilizes a combined quantitative and qualitative survey 
design. 

III. RESULTS 


A. The WLP Student Survey 

The results for the WLP Student Survey were calculated separately for the fall 2010 and spring 2011 
semesters. For both semesters, students reported that their most important consideration when choosing to 
participate in a blended program was the flexibility of being able to complete assignments any place/any 
time (Mfall= 4.09; sd= 1.03; Mspring=3.87, sd= 1.26), closely followed by the convenience of not having 
to come to campus as often (Mfall= 3.95; sd= 1.13; Mspring=3.73, 5(7=1.38). Students in the fall semester 
reported that, on the average, their experiences in blended courses were somewhat worse than in 
traditional f2f courses (47=13.25, sd= 7.12). Students in the spring semester reported that, on the average, 
their experiences in blended courses were neither better nor worse than in f2f courses (M= 17.28, 
5(7=4.24). Students from both cohorts reported that they perceived the workload in blended courses to be 
somewhat heavier compared to f2f (A7fall= 2.31, sd= .68; A7spring=2.28, 5(7=.87). Both cohorts reported 
that overall, they liked the blended courses at the university (Mfall= 21.6, sd= 2.89; Mspring= 19.86, 
sd=4.08). Finally, both cohorts reported that on the average, they found specific components of blended 
courses to be moderately helpful (Mfall= 41.5, sd= 8.10; Mspring= 42.36, sd=6.73). 

A regression analysis was conducted to test predictors of students’ satisfaction with blended learning 
courses. Because the distribution of scores on the variable measuring overall satisfaction with the blended 
course was negatively skewed, this variable was transformed to achieve an approximation of normalcy for 
the purpose of conducting a regression analysis. The results for both cohorts indicate that the factor 
measuring students’ rating of specific course components’ helpfulness is not a significant predictor of 
students’ overall satisfaction with a blended course. Instead, students’ rating of the sufficiency of 
resources for blended learning provided by the university (which both cohorts rated, respectively, as 
moderate) emerged as the strongest predictor of students’ satisfaction (7?fall=-.674 p=.00; 7?spring=-.624, 
/?=.00). This variable accounted for 60% of the variability in students’ satisfaction during the fall 
semester and for 38% of the variability in students’ satisfaction during the spring semester. 

A content analysis of students’ narrative responses on the survey revealed that students found the 
following to be the most effective aspects of blended learning courses: 

1. Flexible scheduling as helpful with time management; 

2. Opportunities for independent work; 

3. Opportunities for interaction with the instructor and with peers; 

4. Good organization of the course and course materials by the instructor; 
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5. Instructor’s sympathetic attitude towards students. 

The following aspects were identified as the least effective aspects of blended learning courses: 

1. The challenge of independent time management; 

2. Fack of opportunities to interact interpersonally with the instructor and with peers; 

3. Confusion regarding assignments and the scheduling of F2F sessions; 

4. Technical difficulties with the FMS or limited computer skills. 

B. The WLP Faculty Survey 

The results for the WFP Faculty survey were calculated separately for each semester. Analysis of 
measures of central tendency indicated that the distribution of scores approximated the normal curve for 
each factor and for total score for both semesters, with the exception of the scores on the Assessment of 
Feaming category for the fall semester, which were slightly positively skewed. The results further 
indicated that, on average and for both semesters, faculty rated the overall quality of their respective 
blended courses as adequate. Table 6 summarizes the results for the total score as well as for each of the 
six categories of the WFP Faculty Survey for each semester. 



Fall 

M/sd 

Spring 

M/sd 

Category 1 - Learner Support and Resources 

9.86 

2.53 

9.36 

2.97 

Category 2 - Online Organization and Design 

10.6 

2.44 

11.9 

2.54 

Category 3 - Instructional Design and Delivery 

16.21 

3.33 

15 

3.63 

Category 4 - Assessment and Evaluation of Student Learning 

9.92 

2.01 

9.72 

2.72 

Category 5 - Innovative Teaching Technology 

8.21 

1.76 

8.09 

2.62 

Category 6 - Faculty Use of Student Feedback 

8.14 

2.59 

8.27 

1.84 

Total score 

59.6 

17.12 

62.36 

14.10 


* All scores fell within the “adequate” range. 

Table 6. The WLP Faculty Survey Mean Category and Total Scores* 


A step-wise multiple regression analysis was conducted to test the relative predictive value of each of the 
categories on the overall faculty rating of course quality. The results for the fall semester revealed that 
faculty ratings on five of the six categories (with the exception of innovative teaching with technology) 
were significant predictors of the overall quality rating of the course. With a 5 value of 3.744 (p=.00 1), 
the category measuring the quality of learner support and resources emerged as the strongest predictor of 
overall course quality score, accounting for 61% of the variance in faculty rating of course quality. The 
second strongest factor, with a B value of 3.20 (p=. 022) was the category measuring the quality of online 
organization and design of the course, accounting for 15% of the variance in faculty rating of course 
quality. The quality of instructional design and delivery (5=1.83, p=.003) and assessment of learning 
(5=1.97, /?=.018) emerged as the next two strongest factors, contributing 14% and nearly 5 %, 
respectively, to the variance in faculty rating of course quality. Finally, the use of student feedback 
(5=1.27, p=.006) emerged as a modest contributor to the overall rating of the course quality, accounting 
for 3% of the variance in faculty rating of course quality. 

The results for the spring semester revealed that faculty ratings on only three of the six categories were 
significant predictors of the overall quality rating of the course. Consistent with the results of the fall 
semester, the quality of learner support and resources emerged as the strongest predictor of overall course 
quality rating (5= 4.10, p=.001), accounting for 75% of the variance in faculty rating of course quality. 
Instructional design and delivery emerged as the next strongest predictor (5=2.23, p=.00) accounting for 
nearly 19% of the variance in faculty rating of course quality. Finally, innovative teaching technology 
emerged as a modest predictor of overall course quality rating (5=1.62, /?=.005), accounting for just under 
2% of the variance in faculty rating of course quality. 
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Item analysis across categories was conducted to identify faculty’s perceptions of the strengths and 
weaknesses of their blended courses. The findings indicated that during the fall semester, faculty gave the 
three highest ratings to items that assess organization of the course: Clarity of the syllabus (M=, 2.5 
sd= .65), clarity of learning objectives (M=, 2.57 sd= .51), goodness of fit between learning activities and 
learning objectives (M=, 2.57 sd= .51). Faculty assigned the lowest 3 ratings to items that assess 
availability of information on online learning and campus resources (M=, 1.57 sd= .64) and to items that 
assess the quality of pedagogy in blended courses: assessing students’ readiness for learning (M=, 1.64 
sd= .63), use of multimedia elements to support different learning styles (M=, 1.64 sd= .63), and 
enhancement of student learning by addressing multiple learning styles (M=, 1.71 sd= .72). Similar 
findings emerged from the spring semester data. Faculty gave the highest ratings to items that assess 
organization of the course (M=, 2.63 sd= .50) and the clarity of the syllabus (M=, 2.63 sd= .50). 
Additionally, faculty also gave a high rating to an item that assesses the degree to which course activities 
develop students’ problem solving skills (M=, 2.54 sd= .52). Consistent with the fall semester data, 
faculty in the spring semester assigned the lowest three ratings to items that assess the degree to which the 
course provides information for online learners’; support (M=, 1.63 sd= .67), the degree to which the 
course provides information on resources for learning (M=, 1.45 sd= .68), and to items that assess the 
quality of pedagogy in blended courses: the degree to which the course enhances student learning by 
addressing multiple learning styles (M= 1.36 sd= .67). 

C. The Focus Groups 

A constant comparative analysis was used to analyze the content of focus group discussions on each 
campus following each semester. The focus groups brought to life the needs of each program, some of 
which were found to be consistent across programs, and others that were unique to a particular program. 
Universal themes that emerged during the focus groups on both campuses over both semesters included 
the need for more faculty development in pedagogy specific to blended instruction and enhanced support 
for students in terms of resources, computer literacy, and organizational and time management skills. 
Regarding the theme of faculty development, there was much discussion on identifying opportunities and 
mechanisms for additional support that would be coordinated and provided by the university’s 
instructional designers. For example, one participant commented: “We’re doing a faculty development 
program where faculty receive hands-on training with Blackboard”. Another participant suggested that 
there should be “more dialogue and sharing [of ideas] among faculty.” 

On the theme of enhanced support for students, the focus group discussions included the type of 
challenges that students face in blended learning courses. For example, several participants highlighted 
the issue of students’ preparedness relative to computer skills in general, and lack of familiarity with the 
Blackboard, the university’s course management system. One participant offered the following example: 
“One of our peeves is when we ask students to post assignments and it would be posted in the body of the 
discussion. [This] reflects a lack of computer tech knowledge”. Another participant suggested that it 
would be “useful to have pre/post-test for entering blended graduate students to self-assess their strengths 
and weaknesses”. The issue of time management was also noted as a challenge for students in blended 
programs. Several participants noted that for students who are taking more than one blended course per 
semester, there is a potential for “conflicts related to scheduling the face-to-face days and online days”. It 
was pointed out that “faculty would like to work more closely within their respective department in the 
scheduling remote classes and making it more convenient for the student.” 

The findings further identified program-specific needs and concerns. For example, during one focus 
group, it was revealed that entering students in a graduate nursing education program needed orientation 
in certain basic computer functions such as disabling pop-up blockers so they could upload attachments in 
Blackboard. Entering students in a graduate computer science program, on the other hand, did not require 
such instruction. However, some computer science students needed greater in-person communication 
with the instructor than had been anticipated. 
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IV. SUMMARY AND CONCLUSIONS 

A. Interpretation of Findings 

This article focuses on indirect measures of blended learning, with an emphasis on the development and 
utility of quantitative and qualitative instruments to gather feedback from students and faculty on blended 
learning during the first year of the university Web Feaming Project. Previous work in the area of blended 
learning assessment focused on assessment of specific components of blended learning from either the 
student perspective [3-10] or the faculty point of view [11-15]. 

This study takes a more comprehensive approach by looking at student and faculty feedback and 
collapsing data across disciplines to gain a deeper understanding of the overall experiences of students 
and faculty in blended programs. Overall, the findings revealed that students in the five pilot programs 
moderately liked their blended courses and that faculty thought their blended courses were adequate in 
terms of quality. The findings further suggested that faculty believed they were doing best with aspects of 
blended courses that were shared by traditional f2f courses such as organization of the syllabus and clarity 
of the learning objectives. Flowever, faculty assigned a lower quality value to course aspects that were 
unique to blended learning, such as the use of specific pedagogy. Possibly, faculty were able to draw on 
their traditional teaching experiences to aid them in designing and delivering specific aspects of the 
blended course, but had difficulties with course components that were unique to blended learning and 
with which they might have had less experience. This discrepancy in skill level has implications for 
designing support services and professional development activities for faculty that target pedagogy for 
blended courses. 

Students and faculty identified the availability of university resources and support for online learning 
(such as library and information resources, academic support services, technical support and student 
services) as important predictors of students’ satisfaction with blended learning and course quality. While 
the value of university resources and support for online learning remained a strong predictor of faculty 
rating of course quality during both semesters (accounting for 61% and 75% of overall course quality 
during the fall and spring semesters, respectively), there was a significant drop in its predictive value on 
students’ satisfaction with their blended courses (from 60% of the variance in the fall to 38% of the 
variance in the spring). Possibly, as students gained more experience with blended learning, they became 
more comfortable with online activities and tasks and thus the availability of university support and 
resources became less central to their overall satisfaction with blended learning. Nevertheless, feedback 
from both semesters suggested that students and faculty believed that there was a need for increased 
university resources and support for online learning as well as an increase in accessibility of resources 
already in place. 

B. Next Steps 

The university has taken a comprehensive approach in utilizing these findings and translating them into 
institutional initiatives along three main dimensions: resources and support for students and faculty, 
professional development for faculty, and expanding the blended learning programs within the university. 

1. Resources and support for students and faculty initiatives 

In response to findings regarding the importance of university resources and support for online learning, 
and in an effort to better communicate to faculty and students the availability of existing resources in the 
institution, the Office of Academic Affairs launched a new blended and online learning website in fall 
2011 (www.liu.edu/online) that includes additional resources on blended/online learning for faculty. The 
new website, still under development, lists and links to all blended and online programs at the institution, 
and will eventually serve as a repository for a variety of blended and online learning resources, including 
a link to the Centers for Student Information supervised by the Offices of Information Technology 
Additionally, the Office of Information Technology will work to increase the availability of both on- 
campus and web-based services during normal business hours, as well as during evenings and weekends. 
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2. Professional development for faculty 

In response to the faculty’s expressed need for training in specific blended learning pedagogies, the 
university is launching three major initiatives: (A) an online faculty training course leading to internal 
certification and delivered within Blackboard to develop instructional competency. The training will 
focus on pedagogical issues including the creation of a blended/online learning community via 
appropriate interactions between and among faculty and students. To aid in this effort, the university has 
purchased licenses for Adobe Connect Pro, a web conferencing software tool, for synchronous 
communications within blended and online courses, and will gradually release the licenses to faculty and 
provide training to ensure appropriate use of the tool; (B) additional online faculty training courses on 
topics such as facilitating online discussions; facilitating group work in online courses; creating and using 
rubrics for assessment of blended/online courses, and using Blackboard and Web 2.0 tools to aid faculty 
in their teaching; (C) faculty access to a dedicated server and training workshops in the development of 
multimedia content, including podcasts. This particular initiative responds directly to the faculty’s 
expressed need to develop expertise in accommodating multiple student learning styles. 

3, Expanding the blended learning programs within the university 

In light of the general favorable response by students and faculty to blended learning programs, the 
university is in the process of launching additional blended courses and programs, as well as some fully 
online programs. Within this initiative, the university is exploring suitable mechanisms for designing 
original blended courses and/or programs as well as for converting f2f courses or programs into a blended 
format. Further, the university is exploring mechanisms to increase support for blended and online 
learning in the areas of admissions, advising, registration, financial aid procedures, etc., within the 
existing university infrastructure. 

C. Strengths and Limitations of the Study 

This evaluation study has several strengths: first, it utilized multiple levels of data collection, thus 
capturing both student and faculty perspectives on blended learning, and highlighting the corresponding 
perspectives and needs of learners and instructors. Second, the combined quantitative/qualitative design 
of the instruments affords a deeper understanding of the student and faculty experience in the blended 
learning programs. Third, the study encompassed feedback on blended learning from various disciplines, 
thus affording the university a comprehensive perspective of blended learning at the institutional level. 
Fourth, the study collected two separate sets of data during two semesters, providing a replication of the 
design and procedures and enhancing the reliability and validity of the findings. 

A major limitation of this study is its relatively small sample size. It is important to note that this was a 
pilot project in its first year with modest enrollment. Additionally, during this first year, as the 
assessment team was developing the data gathering procedures, students and faculty were acclimating to 
both the FMS and the external web-based surveying platforms that were used for assessment of blended 
learning, which may explain the relatively low response rate to the assessment instruments. The WFP 
assessment team will be working with faculty next year on strategies to increase the response rates to both 
the student and faculty surveys. Hopefully, as the university expands its blended learning programs, there 
will be opportunities to collect data from a larger number of students and faculty. Furthermore, while 
collecting data across disciplines is useful in understanding the overall institutional needs of the 
university as it develops its blended learning programs, the small sample size has prevented a more 
specific evaluation of departmental outcomes and needs. As the university expands its blended learning 
program, future assessment can focus on discipline-specific feedback for maximizing program 
effectiveness. 
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